The Effectiveness of Direct Feedback of Faculty and Standardized Patient after Medical Students’ Clinical Performance Examination / 의료커뮤니케이션

BACKGROUND: Clear and proper feedback is necessary to decrease the gap between knowledge and practice. Feedback can improve clinical competence of medical students up to expertise level.METHODS: A total of 180 4th-year students of Daegu-Kyungbuk consortium area participated in the scheduled feedback program of clinical performance examination on August 26th in 2017. Among them, 167 students filled out the questionnaire on helpfulness of the direct observation and feedback of faculty and standardized patient (SP), helpfulness for history taking (Hx), physical examination (PE), and patient-physician interaction (PPI), and any change of clinical competence and clinical reasoning of the students preand post-feedback. All the responses were measured on 5-point Likert scale.RESULTS: Among the total students, 88.6% responded that direct feedback at the practice is helpful, particularly 95.2% of faculty and 76.6% of SP. A 37.3% answered that it was helpful for all the categories of feedback, but only 25.3% for PE and 24.7% for Hx. A 56.3% responded that feedback is helpful for both disease questions and counseling practice. Mean score of self-assessment by the students increased from 2.52 to 3.36 for Hx, 2.30 to 3.24 for PE, 2.46 to 3.33 for clinical reasoning, 2.84 to 3.59 for PPI, and 2.44 to 3.28 for overall competence. And all these differences were found to be statistically significant (P <0.001)CONCLUSION: A majority of students responded that the direct feedback from faculty and SP was helpful at the clinical practice of Hx, PE, and PPI, and increased the level of clinical competence.

Responses of Medical Students to Using Smartphone Video at Clinical Performance Examination / 의료커뮤니케이션

BACKGROUND: There are practical difficulties to show exact clinical symptoms such as seizure to medical students at Clinical Performance Examination (CPX). We developed a new CPX case of child's seizure on video using smartphone.METHODS: A total of 356 4th-year students of five universities in Daegue-Gyeongbuk and Gyeongnam area took the clinical skill examination from June 13th to 17th in 2016. Among them, 72 students took the new CPX case in June 15th and 71 students filled out the questionnaire on whether the new CPX with smartphone video is helpful, authentic, difficult, and necessary for other CPX. All the questions were measured on 5-Likert scale.RESULTS: Mean score of the new CPX was 57.1, lower than the mean scores of the other 11 CPX cases, 62.8. For the question “Smartphone videos helped to solve the problem”, 45 students (63.4%) answered ‘Very much’. For the question “Is it realistic compared to other questions?” 30 students (42.3%) and 25 students (35.2%) answered ‘Very much’ and ‘Much’. For the question “Is it difficult compared with other questions?” 18 students (25.4%) and 26 students (36.6%) answered ‘Very much’ and ‘Much’. As for the question “I would like to have more tests using smartphone video”, 26 students (36.6%) answered ‘So and so’.CONCLUSION: A majority of students responded that video presentation was helpful and authentic to figure out the CPX, whereas they assessed smartphone video was more difficult compared with other CPXs. Further, students were negative toward using smartphone video for the other CPXs.

Selection of Tasks for Assessment of Pharmacy Clinical Performance in Korean Pharmacist Licensure Examination: Results of an Expert Survey

OBJECTIVE: As the demands of pharmacist's role and quality performance have increased, the verification of pharmacist's ability has been required. In this study, we aimed to select appropriate items for assessment of pharmacist's knowledge, attitude and performance. METHODS: Based on the pharmacist job analysis, we selected duties and tasks in consideration of applying pharmacy practical examination through brainstorming of internal researchers and group discussion with experts. Survey was conducted to evaluate the tasks according to the criteria detailed below: Realistic, Understandable, Measurable, Behavioral and Achievable (RUMBA). The subjects included professors at colleges of pharmacy and instructors of institutional or community pharmacy settings. RESULTS: Nine duties including 41 tasks were drawn for the survey through primary internal researchers. Of the 90 respondents, 95.6% were professors or preceptors who was engaged in practical training, and 62.2% had more than five years of practical experience. As a result of survey and discussion with expert panel, selected seven duties were selected as followings: ‘Patient (customer) reception’, ‘Drug preparation and distribution’, ‘Patient care’, ‘Administration’, ‘Patient counseling’, ‘Non-prescription medication counseling’, and ‘Provision of drug information’. The final 20 tasks from seven duties were chosen to assess skills that a pharmacist should be able to perform. CONCLUSION: This is the first study to select the items that can be included in pharmacist practical examination in the future, based on the RUMBA criteria. As a next step, it is necessary to study how to implement these items.

Perception survey on the introduction of clinical performance examination as part of the national nursing licensing examination in Korea / 보건의료교육평가

PURPOSE: The purpose of this study was to analyze opinions about the action plan for implementation of clinical performance exam as part of the national nursing licensing examination and presents the expected effects of the performance exam and aspects to consider regarding its implementation. METHODS: This study used a mixed-methods design. Quantitative data were collected by a questionnaire survey, while qualitative data were collected by focus group interviews with experts. The survey targeted 200 nursing professors and clinical nurses with more than 5 years of work experience, and the focus group interviews were conducted with 28 of professors, clinical instructors, and nurses at hospitals. RESULTS: First, nursing professors and clinical specialists agreed that the current written tests have limitations in evaluating examinees' ability, and that the introduction of a clinical performance exam will yield positive results. Clinical performance exam is necessary to evaluate and improve nurses' work ability, which means that the implementation of a performance exam is advisable if its credibility and validity can be verified. Second, most respondents chose direct performance exams using simulators or standardized patients as the most suitable format of the test. CONCLUSION: In conclusion, the current national nursing licensing exam is somewhat limited in its ability to identify competent nurses. Thus, the time has come for us to seriously consider the introduction of a performance exam. The prerequisites for successfully implementing clinical performance exam as part of the national nursing licensing exam are a professional training process and forming a consortium to standardize practical training.

The Effects of Preclinical Clinical Performance Examination on Nursing Students' Confidence in Nursing Skills and Critical Thinking Competence / 한국간호교육학회지

PURPOSE: The purpose of this study was to investigate the effects of preclinical clinical performance examination (CPX) on nursing students' confidence in their nursing skills and critical thinking competence. METHODS: The design of this research was one-group pretest-posttest, and the participants were 112 nursing students. The preclinical CPX consisted of a clinical examination, patient-nurse relationship, oral test of related knowledge, written test of the nursing process, and debriefing using comprehensive scenarios based on real patient cases. The confidence of nursing skills consisted of an 8-item NRS and the critical thinking competence consisted of a 12-item 4-point scale developed by researchers and measured in both the pretest and posttest. The collected data were analyzed using paired t-tests, ANOVA, and Pearson correlation coefficients. RESULTS: The score for confidence in nursing skills (t=10.60, p<.001) and that for critical thinking competence (t=7.03, p<.001) increased significantly after preclinical CPX. CONCLUSION: This study showed that preclinical CPX was effective in improving nursing students' confidence in their nursing skills and critical thinking competence. Therefore, preclinical CPX is expected to be utilized in nursing practice education. Additional studies including those on control groups are recommended to compare differences between the preclinical CPX group and control group.

A school-level longitudinal study of clinical performance examination scores / 한국의학교육

PURPOSE: This school-level longitudinal study examined 7 years of clinical performance data to determine differences (effects) in students and annual changes within a school and between schools; examine how much their predictors (characteristics) influenced the variation in student performance; and calculate estimates of the schools' initial status and growth. METHODS: A school-level longitudinal model was tested: level 1 (between students), level 2 (annual change within a school), and level 3 (between schools). The study sample comprised students who belonged to the CPX Consortium (n=5,283 for 2005~2008 and n=4,337 for 2009~2011). RESULTS: Despite a difference between evaluation domains, the performance outcomes were related to individual large-effect differences and small-effect school-level differences. Physical examination, clinical courtesy, and patient education were strongly influenced by the school effect, whereas patient-physician interaction was not affected much. CONCLUSION: Student scores are influenced by the school effect (differences), and the predictors explain the variation in differences, depending on the evaluation domain.

Impact of Clinical Performance Examination on Incoming Interns' Clinical Competency in Differential Diagnosis of Headache / 가정의학회지

BACKGROUND: In Korea, clinical performance examination (CPX) has been included in license examination for medical doctors since 2009 in order to improve clinical performance of medical students. This study aimed to evaluate the contribution of CPX to medical education. METHODS: Clinical competency in the differential diagnosis of secondary headache was compared between the incoming interns in 2009 unexposed to CPX and the incoming interns in 2010 exposed to CPX, using the data of patients who visited the emergency department due to headache (181 patients seen by 60 CPX non-exposed interns and 150 patients seen by 50 CPX-exposed interns). We obtained the data by reviewing electronic medical records and nominal lists of doctors. Clinical competency was assessed by sensitivity and specificity between the diagnostic impression by interns and the final diagnosis. The association between CPX exposure and clinical competency in secondary headache diagnosis was evaluated using multiple logistic regression analysis. RESULTS: When we assessed clinical competency on the basis of all listed diagnostic impressions, sensitivity and specificity were 67.9% and 80.0%, respectively, for headaches seen by CPX-exposed interns, and 51.7%, and 71.7%, respectively, for headaches seen by CPX non-exposed interns. Multivariable adjusted logistic regression analysis showed exposure to CPX was not associated with increased competency for identifying secondary headache. CONCLUSION: Exposure to CPX as a part of the medical license examination was not effective for the improvement of clinical competency of interns in identifying secondary headache.

Changes in medical students' patient-centeredness attitudes by implementation of clinical performance examination / 한국의학교육

PURPOSE: From 2009, the Korean Medical Licensure Exam implemented a clinical performance examination (CPX) that highlighted the importance of good patient-physician relationships in medical education. This study aimed to examine changes in medical students' attitudes before and after implementation of the CPX in their medical education curriculum. METHODS: In 2006 and 2009, 236 fourth-year medial students of C College of Medicine took the Patient Practitioner Orientation Scale (PPOS) by Krupet et al. (2000), which measures patient-centered attitudes in patient-physician relationships. The data were analyzed by independent t-test and two-way analysis of variance using SPSS 21.0. RESULTS: The PPOS scores of year 2006 students were 3.88+/-0.25 for males and 3.98+/-0.38 for females. For year 2009 students, males scored 3.81+/-0.42 and females scored 4.01+/-0.48. All students had higher Care scores than Share scores (2006: Share, 3.67+/-0.47 vs. Care, 4.19+/-0.51; 2009: Share, 3.56+/-0.34 vs. Care, 4.18+/-0.53). There were significant differences in PPOS and Share scores by gender before and after the CPX. With regard to Care scores, female students' scores tended to rise and males' scores declined over time. CONCLUSION: An educational program is needed for students to foster patient-centered attitudes, but gender differences should be taken into account.

Improvement in Clinical Performance of Interns and Residents through Clinical Skills Assessment of the Korean Medical Licensing Examination / 한국의학교육

PURPOSE: To evaluate the clinical performance through the Korean Medical Licensing Examination clinical skills assessment (KMLE CSA) this survey was done. METHODS: A survey of 130 interns and residents (46 applicants and 84 non-applicants for the KMLE CSA) at a university hospital in Seoul was conducted in January and February 2012. The data were gathered using a structured and self-administered questionnaire. For the items that assessed the clinical performance of these subjects, we selected 15 items that are mostly frequently used by Delphi's technique, and difficult procedural skills based on the results of medical students' performance. We also used subcomponents of the clinical problems test of the KMLE CSA. RESULTS: The total score on the KMLE CSA improved by 1.33 points (a perfect score is 10), 1.49 points for procedural skills, and 0.84 points for clinical problems by multiple regression analysis. The variables that influenced clinical skills were sex (females had 0.86 more points than males), experience in military or public services (1.04 points higher than persons without experience), and type of school (graduates of medical school had 1.41 more points than graduates of professional graduate school). CONCLUSION: Implementation of the KMLE CSA improved the clinical performance of medical graduates.

Emerging tasks of specialty certifying examination: educational measurement considerations

Medical specialty systems were launched in 1951 by the National Medical Services Law. The following year, the specialty certifying examination had implemented in the form of portfolio evaluation. A paper-and-pencil type examination was implemented in 1960, and the 55th examination was carried out in January 2012. Currently, 26 specialties are represented, and the overall pass rate is over 90%. The examination consists of a step 1 paper-and-pencil test and step 2 skills test. In the step 1 test, the test items are multiple choice questions and short answer questions. Clinical performance examination is partially applied to the step 2 test. To cope with changes in the social situation and the growth of medical services, developmental changes are needed in the specialty certifying examinations. Performance assessment is an alternative worth considering. CPX should be a major part of the skill test. A computer-based test should be introduced as soon as possible, and it could eventually be developed into an adaptive test.

Effect of Emotional Intelligence on Patient-Physician Interaction Scores of Clinical Performance Examination / 한국의학교육

PURPOSE: The patient-physician interaction (PPI) is a critical part of the clinical encounter. Recent studies have emphasized the importance of the emotional intelligence (EI) of physician in the PPI. Despite emphasizing the EI, previous studies offer limited evidence regarding the effect of a student's EI on the PPI. The purpose of this study is to explore the differences in EI depending on the demographics of medical students and the correlation between EI and PPI scores. METHODS: The sample was 85 fourth-grade medical students. Prior to taking a 12-station clinical performance examination, the students completed questionnaires on their own perception of the EI, which included 5 domains and 50 items. The tool that was used to assess the level of EI was Moon's modified version of the EI test for adults. We investigated differences in EI depending on the demographics of medical students by ANOVA and noted a correlation between EI and PPI scores by stepwise multiple regression analysis. RESULTS: This study found that females or graduate entry students have higher EI scores and that 25 to 30-year-old students have higher EI scores than aged under 25 years. The PPI scores correlated positively with total EI scores (r=0.32) and 2 subdomains (perception and expression of emotion, r=0.26; empathy, r=0.33). Two subdomains were the best predictors of PPI score (R2=0.171). CONCLUSION: EI correlates significantly with PPI score and affects it. We conclude that EI is a key influence of the PPI. Further research is required to explore whether this is a consistent effect.

The Change of CPX Scores according to Repeated CPXs / 한국의학교육

PURPOSE: Most medical schools have held clinical skills training programs recently. Despite these educational endeavors, few studies have attempted to address the effect of clinical skills assessments on clinical performance. This study investigated whether repeated experiences with the examination improved medical students' history-taking, physical exams, and patient-physician interactions (PPIs). METHODS: The subjects of the study were 101 4th-year medical students who participated in the clinical performance examination (CPX) 3 times. They completed their core clerkship before acquiring the first CPX scores; we tracked down the scores of three sets of CPX for 3 subdomains (history taking, physical exam, and patient-physician interaction) and investigated the changes in these scores. Additionally, we classified the research subjects into 3 groups by total CPX score-higher (upper 30%, n=30), intermediate (medium 40%, n=40), and lower (lower 30%, n=30)-and compared the curves for each group. RESULTS: Significant improvements were made on history taking and physical exam (F=130.786/237.358, p<0.01), while proficiency on the PPI declined (F=17.621, p<0.01). Additionally, scores in all levels improved continuously on history taking and physical exam, while students of the high and low levels experienced a sharp decline on the PPI (F=11.628, p<0.01). CONCLUSION: Improvement in the history-taking score reflects an accumulation of clinical knowledge and clinical exposure. Improvement on the physical exam score is affected by repeated practice on similar or identical cases and receipt of feedback. That PPI can deteriorate might be an effect of one's negative experience in a clinical clerkship.

The School Effect on the Reliability of Clinical Performance Examination in Medical Schools / 한국의학교육

PURPOSE: The purpose of this study is to test the reliability of the clinical performance examination (CPX) using Generalizability theory (G-theory). Through G-theory, the effects of not only students and tasks but also the school will be analyzed as primary sources of error, which can affect the interpretation of the reliability of the CPX. METHODS: One thousand three hundred nineteen students from 16 medical schools that participated in the Seoul-Gyeonggi CPX Consortium 2008 were enrolled. In our research design, we suppose that student is nested within school and crossed with task. Data analysis was conducted with urGenova. RESULTS: According to our analysis, the percentage of error variance was 6.2% for school, 14.9% for student nested within school, 14.4% for task, and 3% for interaction between school and task. An effect of school on students was observed, but the interaction between task and school was insignificant. When student is nested within school, the universe score decreased and the g-coefficient was less than the g-coefficient of the p x t (p: studentm, t: task) design. CONCLUSION: The results show that generalizability theory is useful in detecting various error components in the CPX. Using the generalizability theory to improve the technical quality of performance assessments provides us with greater information compared with traditional test theories.

Introduction and Administration of the Clinical Skill Test of the Medical Licensing Examination, Republic of Korea (2009) / 보건의료교육평가

The first trial of the clinical skill test as part of the Korean Medical Licensing Examination was done from September 23 to December 1, 2009, in the clinical skill test center located in the National Health Personnel Licensing Examination Board (NHPLEB) building, Seoul. Korea is the first country to introduce the clinical skill test as part of the medical licensing examination in Asia. It is a report on the introduction and administration of the test. The NHPLEB launched researches on the validity of introducing the clinical skill test and on the best implementation methods in 2000. Since 2006, lists of subjects of test items for the clinical skill test has been developed. The test consisted of two types of evaluation, i.e., a clinical performance examination (CPX) with a standardized patient (SP) and objective structured clinical examination (OSCE). The proctor (medical faculty member) and SP rate the examinees' proficiency for the OSCE and CPX respectively. Out of 3,456 applicants, 3,289 examinees (95.2%) passed the test. Out of 167 examinees who failed the clinical skill test, 142 passed the written test. This means that the clinical skill test showed characteristics independent from the written test. This successful implementation of the clinical skill test is going to improve the medical graduates' performance of clinical skills.

The Accuracy of Standardized Patients' Rating according to the Order of Examinees in Clinical Performance Examination / 한국의학교육

PURPOSE: The purpose of this study is to evaluate the accuracy of standardized patients'(SP) rating according to the order of examinees in clinical performance examination. METHODS: In the clinical performance examination which was administered in 2005 at Seoul National University College of Medicine, each SP evaluated 16 students consecutively. For all 16 SPs(2 SPs per station), accuracy of rating was evaluated by comparing the individual records of each SP to the 'recording keys' made by two SPs from reviewing a video recording of performances by examinees. RESULTS: The average number of items incorrectly rated by SP was 3.8(range, 0~12), 2.8 in female SPs and 4.8 in male SPs(p<0.001). No statistical correlation was observed between the number of errors and the order of examinees(p=0.843). Even after stratification by gender or age of the SPs or domains of examination, the number of items incorrectly rated did not differ significantly according to the order of examinees. CONCLUSION: An increase in SP's rating error with time after the start of examinations was not observed within the 16 consecutive encounters in clinical performance examination. The effect of SP's fatigue on the accuracy of simulation as an examination progresses remains to be studied.

Item Analysis of Clinical Performance Examination Using Item Response Theory and Classical Test Theory / 한국의학교육

PURPOSE: The objectives of this study were: 1) to analyze Clinical Performance Examination(CPX) items using item response theory(IRT) and classical test theory(CTT) and 2) to discuss how to apply and interpret these results in order to improve the quality of CPX items. In addition, we intended to explore statistical procedures in order to merge examination data from several different medical schools. METHODS: The subject of the study was the 2005 CPX examination data from 10 medical schools located in Seoul and the Kyunggi province. For merging data from ten different medical schools, Levene's test for homogeneity of variances was used. Homogeneous group selection was conducted based on ANOVA or Kruskal-Wallis' test and Tukey's multiple comparisons appropriately. The generalized partial credit model was applied to analyze polytomous items and the 2-parameter logistic model was used to analyze dichotomous items. RESULTS: Data from 8 medical schools were incorporated into the analysis. The result of the discrimination index by IRT was different from that of CTT in both polytomous and dichotomous items. Discrimination index from IRT tended to be lower than that of CTT. Difficulty index of dichotomous items of two models was correlated well with each other. However, for polytomous items, IRT model provided more information than CCT. CONCLUSION: We discovered that the CPX items were mostly easy in terms of difficulty index, and the result from IRT and CCT model did not correlated well in the discrimination index. IRT may provide more detailed information for polytomous items, but the checklist and criteria of scoring system should be cautiously reviewed.

The Effects of Information Sharing between Students on Results of Clinical Performance Examination / 한국의학교육

PURPOSE: This study was designed to investigate the effects of information sharing between students on results of clinical performance examination (CPX). METHODS: 143 third-year students, who completed their core clinical clerkship, were randomly assigned to one day of a 3-day, 6 station CPX. Station checklists, scored by the SP(s), consisted of history taking, physical examination, information sharing, clinical courtesy and patient-physician interaction. We compared the CPX station scores, CPX domain scores, self assessment scores and GPA of the three groups tested on different days with ANOVA, and analyzed the differences in CPX scores, after controlling for GPA, with ANCOVA. The correlations among the CPX total and domain scores, self assessment and the GPA of the 3rd year students were calculated. Data regarding information sharing between students and self-assessment were obtained through student questionnaires. RESULTS: CPX total scores, self assessment scores and GPA of 3rd year students from day-2 were significantly higher than scores from day-1 or day-3. Checklist analysis showed that PE, CC and PPI scores were significantly higher on day-2, but Hx and IS scores were not significantly different. Results from questionnaires showed that 43% of students shared evaluation information. And CPX total scores had little correlation with self assessment scores. GPA of 3rd year students showed a similar pattern with the CPX totalscore. CONCLUSION: There is information sharing occurring between students during clinical performance examinations. Although there are significant differences in CPXscores for the treatment group, it seemed that CPX scores are not affected by information sharing between students. Reliability of information sharing from other students is low. The generalization of this study should be interpreted carefully.

Experience with Clinical Performance Examination using Standardized Patients at Gachon Medical School / 한국의학교육

PURPOSE: This study aims to abstract the differences of scores between the clinical instructors and standardized patients (SPs) in a clinical performance examination (CPX) using SPs and to correlate the concordance between the evaluation scores and the school records. METHODS: The CPX was administered in 2003 to a total of thirty-six fifth year medical students at Gachon Medical School. The examination consisted of four cases, and four stations were duplicated, each requiring a total of 7.5 minutes per station. Evaluation of the student' s performances was conducted by both clinical instructors and SPs using a formatted checklist. Results were analysed by t-test, agreement rates, and Pearson correlation. RESULT: The mean scores given out by the clinical instructors and the SPs for the newly developed case were significantly different, while those scores for pre-existing cases were not in the old cases. The correlation coefficients between these two evaluation groups were relatively high. And agreements between the two evaluation groups were 0.37~0.72. The mean scores among clinical instructors were not significantly different, but the correlation coefficients and agreement rates were relatively high. The correlation between the evaluation scores and school records did not correlate significantly. CONCLUSION: It is concluded that the CPX is a useful tool to measure the students' essential competences in areas of knowledge, skills and attitude during the subinternship stage. In conducting a successful CPX, it is crucial to reconsider the recycling of cases and the selection and training of SPs aside from the development of an objective checklist.

The Appropriateness of using Standardized Patients' (SPs) Assessment Scores in Clinical Performance Examination (CPX) / 한국의학교육

PURPOSE: The purpose of the study is to analyse the appropriateness of using standardized patients' (SPs) assessment scores of medical students in clinical performance examination (CPX). METHODS: 110 fourth year medical students in year four were divided into two large groups (group A and group B). Each group of students performed four different cases and overall, eight cases were tested. The examination were done by professors and the SPs using the same examination paper. Test scores were analysed per station and per each test domain. The differences between the two examiners were studied using paired t-test. RESULTS: There were significant differences in the scores given out by the professors' and the SP' s in four out of the eight stations. According to the examination results of each domain, four cases showed significant differences in the history taking category, three cases in the physical examination, one case in the information sharing, four cases in clinical courtesy, and two cases in physician-patient relationship category. Cronbach' s alpha scores of all stations were over 0.6, reflecting that the test items were appropriate for the examination. CONCLUSION: From the result of this study and according to many preceding studies, using SPs' evaluation scores in CPX is appropriate. Some limitation of this study and ideas for improvement in using SPs in CPX are suggested.

Comparison of the Evaluation Results of Faculty with Those of Standardized Patients in a Clinical Performance Examination Experience / 한국의학교육

PURPOSE: To compare the evaluation results of faculties to those of Standardized Patients (SP) participating in a Clinical Performance Examination (CPX) administered at Ewha Womans University College of Medicine. METHODS: The CPX was taken by 77 fourth year medical students. Cases and checklist were developed by the medical school consortium in capital area. Six cases were used and 24 SPs participated and evaluated the students' performances. The whole session was recorded on videotapes so that 6 medical school faculties could analyze and evaluate the students' performances as well. The results were compared and analyzed by SPSS package. RESULTS: The agreement between the faculties and the SPs was relatively good (r=0.79), but not good enough. In every case, SPs gave higher marks than did the faculties. Clear disease entity cases like "hepatitis" and "anemia" showed better agreement than obscure clinical contexts such as "bad news delivery". Better agreement was seen in the items of physical exam category (r=0.91), but the agreement was very poor in the items of doctor-patient (Dr-Pt) relationship category (r=0.54). The construction of checklist and the character of each evaluation item should influence the differences. CONCLUSION: More detailed guidelines and clear/specific evaluating items are necessary to improve the agreement rate. In certain categories like physical exam and brief history taking, the SP' s evaluation can replace the faculties', but for complex contexts like Dr-Pt relationship.